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This paper describes the process used to design and 
evaluate a computer-based tool that guides the development of lesson 
plans. The tool, PLANalyst, contains a small expert system with a 
human face that analyzes lessons and gives specific feedback f r 
improvement. The process used to create PLANalyst can be generalized 
to the development of tools to support any complex intellectual task. 
PLANalyst runs on Macintosh computers and was constructed with 
HyperCard. The tool allows easy creation and editing of lessons and 
provides printed out lesson plans that encourage lessons based on 
sound instructional principles and realistic estimation of time. It 
also provides a common language and a tool for team teaching and 
instructional development. The model on which it is based most 
closely resembles the nine events of instruction of R. M. Gagne and 
others. Preparation screens and event screens are illustrated, and 
the expert system is described. Field tests with close to 200 college 
students have supported the utility of the tool. The final version, 
still- in development, will incorporate a database of useful teaching 
techniques drawn from ERIC citations. Six figures illustrate the 
screens. Two appendixes contain event categories and an evaluation 
form. (Contains 7 references.) (SLD) 
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Introduction 

While mcst graduate programs in educational technology focus on the design of instructional 
materials in a wide range of media, the most common delivery mode of education and 
training is still the instructor-led course. Regardless of how much CBT or multimedia 
experience a designer may have, he or she is often called upon to put together a workshop or 
presentation quickly. To enable teachers and trainers to draft and edit lesson plans 
efficiently and to share them with others in a consistent format, a tool called PLANalyst was 
developed. 

This paper describes the process used to design and evaluate a computer-based tool that 
guides the development of lesson plans. The tool, PLANalyst, contains a small expert system 
with a human face which analyzes lessons and gives specific feedback for improvement. The 
process used to create PLANalyst can be generalized to the development of tools to support 
any complex intellectual task. 

PLANalyst runs on Macintosh computers and was constructed with HyperCard. No 
knowledge of HyperCard is required other than familiarity with the graphical user interface 
conventions of mouse and menu use. 

The Problem 

What makes planning a class or workshop or presentation a challenge? What gets in the way 
of good instruction? Novice instructors tend to try to pack too much information into the 
time allotted, don't provide enough opportunities for distributed practice and feedback, forget 
that many learners need to be systematically motivated by having the relevance of the 
content made clear, neglect to orient the learners to the material and activate their prior 
knowledge, and stick to a limited number of tried and true teaching strategies. Seasoned 
instructors do all of these things, too, though less often. 

These mistakes can be avoided by devoting enough effort to the planning of a lesson. As with 
anything else, this kind of planning effort will happen more in the real world if the task can 
be made easy, efficient, and (dare we hope) enjoyable. 

PLANalyst was developed to support the lesson design process. It... 

• allows easy creation and editing of lessons: changing times, adding and deleting 
activities, rearranging the sequence, etc. 

provides clearly printed out lesson plans which show minute by minute how the 
lesson is to be presented. 

encourages users to base their lessons on sound instructional principles and to 
realistically estimate their time requirements. 

• contains a built-in expert system that can evaluate lesson plan and suggest 
changes. 

• provides a common language and tool for team teaching and instructional 
development. The work of designing a lesson can be spread out over multiple 
authors. PLANalyst allows events from separate lesson plans to be merged into 
one. 
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can be used to promote consistency across classes with multiple sections and 
instructors. 

can be used to observe, record, document and evaluate teaching by principals, 
teacher educators and training managers. 

Steps in Creating an Electronic Performance Support Tool. 

The process of designing a tool like PLANalyst follows a path similar to that performance 
technologists use to develop other problem solutions. 

1. Define the problem. 

2. Develop solution specifications. 

3. Develop the interface A 

4. Develop the knowledge base. 

Across all of these steps is a constant process of user testing and refinement as needed. 

The problem definition in this case included the assumptions that the target users of the 
tool... 

are sometimes unsure of the necessary ingredients of a successful lesson 

often forget to include those elements they know to constitute good instruction 

often err in the amount of time allocated to instructional activities 

need to be able to communicate their instructional intentions clearly to others on 

paper 

• see lesson planning as an onerous chore to be done with minimal detail and all in 
one draft 

would benefit by exposure to teaching strategies they might not normally use 

The solution specification included the following characteristics of the tool. The user should 
be able to... 

generate lesson ideas quickly and refine them later 
rearrange the order of lesson activities easily 

consider time requirements at every point while drafting the lesson, 
create attractive printouts in various forms 

• justify the instructional purpose served by every instructor act. 

access specific suggestions for teaching strategies with pointers to published 
articles and other documents 

Design 

PLANalyst is built on the assumption that a presentation or lesson can be divided into 
distinct chunks of one minute or more in duration, and that each chunk should serve some 
instructional or practical purpose. PLANalyst is intended to be equally useful to experienced 
and novice presenters or teachers, though meeting this design goal has proven to be a 
challenge. 
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The instructional design model that PLANalyst is based on most closely is that of Gagne's 
nine events of instruction (Gagne, Briggs & Wager, 1992). That model is most appropriately 
used for direct instruction ...the teacher-centered, objective-driven approach used most often 
in the training of adults. Additionally, suggestions summarized from the adult training 
literature and professional practice by Powers (1992) were incorporated. To widen the 
applicability of the tool to teachers of children, models of indirect instruction and cooperative 
learning were added to the rule base. These models were derived from work by Johnson & 
Johnson (1991), Joyce, Weil & Showers (1992), and Slavin (1983). 

In one fundamental way, however, PLANalyst runs contrary to the standard instructional 
systems design approach: it allows one to begin the process of lesson design by listing 
possible activities rather than proceeding deductively from objectives. This activities-first 
approach is in keeping with how teachers conceptualize the lesson planning process 
(Sherman, Driscoll & Klein, 1994). There is nothing to prevent an instructional designer 
from using PLANalyst by specifying objectives first and going on to draft learning activities. 

There are three major parts to the PLANalyst tool: 

preparation screens, 
event screens, and 
an expert system. 

On Preparation Screens, the lesson designer describes the learners, the content, the 
instructional context, and practical considerations such as room arrangements and materials 
needed. This description can be printed out when the plan is finalized to serve as a reminder 
before the lesson is delivered. There are three such screens in the present version of 
PLANalyst. The presence of fields and checkboxes requiring input reminds the user to 
analyze the learners and subject matter before rushing headlong into design. 
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PLAN - EDTEC 470 Week 1 




PLANalyst 



A Lesson Design Too! 



Lesson Name 



470 Week 1 



Authors 



Bernie Dodge et al 



Content ■ 



Jamacha Elementary . Labwith15LC H's. Student 
teachers in second semester of block. 



2:15 PM to 4:42 PM 



♦ o 



Gettioq Started 
Fill out survey 
Read syllabus 
About 470 (Slide 1) 
Importance (Slide 2) 
Information Aye (Slides 3-4) 
Ed Tech (Slides 5-6) 
Objectives (Slides 7-10) 
Describe Readinys 
Describe Assignments 
Q &. A about course 
Break 

Intro to Macintosh 
Macintosh Tour 
Debrief Mac Tour 
Introduce HS 2000 
Show High School 2000 
Debrief HS 200CL 
For Next Week^T 



To describe each event, the user completes an Event Screen like the one below. 




PLAN - EDTEC 470 Week 1 



Introduce HS2000 



4:11 to 4:21 



Motiuation 
• Arouse Uncertainty 



Present something that seems to contradict vhat the 
learners already know. (Be sure to allow the 
learners to resolve the uncertainty *^ om * point.) 



Duration ■Description 



Ask then to imagine this year's second graders as seniors in the year 2003. 
V/ith the technology that will be available then, how would you teach 
Shakespeare? Vhat will the computers took like? Vhat will the classroom 
look like? 

Ask them to think individually , then brainstorm in pairs, then share list. 





Each event is labeled with a short name (Introduce HS 2000, in this example). A 
hierarchical pop-up menu allows the user to tag the event with one of close to 70 identifiers 
which describe the instructional purpose of the event (in this example "Arouse Uncertainty", 
one of 10 items under the "Motivation" category was chosen. 
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Introduce HS 2000 



4:11 to 4:21 



Medium 



Motivation < 



Objectiues 

Prerequisite Knowledge 
Information Presentation 
Practice & Feedback 
Testing 

Enrichment & Remediation 
Other 

10 OK l*t V 



Nouelty 



fliouse Uncertainty 



Ask th»m to think individually , th* 



1 



Inuoke Mystery 
Cite Model 

Establish Future Releuance 
Establish Past Releuance 
Raise Confidence 
Establish Reward 
Establish Credibility 
Establish Need 



Ith* 



^■Xiok to \\ 

•,m stock.-.; 



The user specifies how long the event will take by clicking on arrow buttons and the time 
span during the which the event will take place is immediately recalculated and displayed at 
the upper right part of the screen. The largest item on the screen is a scrolling field in which 
notes on what to do at this point in the lesson can be written. Users can navigate through the 
let >son by clicking on the arrow buttons in the lower right part of the screen, or by clicking on 
the "Return to Main Menu" button. 



On the Main Menu, the list of all events in the lesson can be seen. » V e can travel to the 
screen describing the event by clicking on its name. Events can be oquenced by dragging 
them up or down through the list. 

With the Expert. System, the user can at any point request advice from a modest knowledge 
base built into PLANalyst. The system scans through the draft lesson plan and looks at the 
amounts of time spent on specific types of events, the sequencing of those events, and 
descriptions of the learner group and content. A rule base of twenty principles is consulted 
which triggers positive comments or corrective feedback as appropriate. The advice is given 
through the persona of Anna Liszt, who is described as an experienced teacher and 
instructional designer. This feedback can be viewed on screen or printed out. 
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Vou'ue prouided some practice within 
the lesson, but I wonder if you'ue 
prouided it soon enough. It seems as 
though there is a Bong stretch of 
information being presented before 
your learners haue a chance to test 
their understanding of it. 



Perhaps you can break up the 
information into subparts and giue 
learners practice at one piece at a 
time? 




Field Testing 

PLANalyst has been used by close to two hundred students at San Diego State University 
during the last two years. About half of those using the tool are pre-service teachers taking a 
course in educational computing; the others are graduate students in instructional design. 
At the end of each semester, students rate each element of PLANalyst and suggest 
improvements. 

Routines in the software (Williams & Dodge, 1993) were used to unobtrusively track data on 
how PLANalyst was used by students. These data include the amouut of time spent in each 
part of the program, the amount and type of editing done, and patterns vf use of the expert 
system. 

A typical sample of the tracking data looks like this: 

2841218642 SO with PLANalyst version .99c3 

2841218648 MBR 

2841218813 AS 8:00 AM 

2841218819 AE 

2841218831 ED Take Roll to 3 

2841218878 EW Take Roll .# Words = 9 

2841218892 AE 

2841218911 ED Importance of Chinese Society to 8 

Which translates into this: 

Wednesday, January 12, 1994 11:24:02 AM 
11:24 Opened stack with PLANalyst version .99c3 
11:24 Menu choice: Brainstorm Events 
11:26 Adjusted starting time to: 8:00 AM 
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11:26 Clicked on Events list 

11:27 Adjusted duration of Event Take Roll to 3 minutes 

11:27 Edited Description of Event Take Roll . Total # Words = 9 

11:28 Clicked on Events list 

11:28 Adjust duration of Event Importance of Chinese Society to 8 minutes 

In the next round of examination of these data now under way, cluster analysis will be used 
to identify categories of usage patterns, and the completed lesson plans of all participants 
will be rated qualitatively and related to these clusters. 

In the earliest trials of the software, user feedback was collected in the form of open-ended 
paragraphs summarizing their opinions, as well as informal interviews with users. In the 
most recent field tests of PLANalyst, a questionnaire was used to get specific feedback on the 
program (Appendix B). The results of testing the most recent version were as follows: 
(N=ll). To simplify the data for a coarse-grained analysis, Agree and Strongly Agree 
responses were lumped together, as were Disagree and Strongly Disagree; Neutral responses 
were left out of the table. 



Statement 


% 


i 




Agre 
e 


Disagree 


The printouts were legible and attractive 


91% 


0% 


The Rearrange Events feature is useful 


91% 


9% 


The Brainstorming Events feature is useful 


91% 


9% 


PLANalyst was easy to use 


91% 


0% 


PLANalyst will be useful primarily for planning special or unusual 


90% 


0% 


events 






The time spent putting a lesson into PLANalyst form would be 


54% 


36% 


well spent 






The program seemed to run slowly 


50% 


27% 


It was difficult to find an appropriate category to describe an 


45% 


36% 


event 






I wanted to include 






There were enough different print out formats provided 


45% 


9% 


The advice from Anna was useful and appropriate 


36% 


45% 


I'd use "Lite" mode more often than the Full mode 


36% 


0% 


PLANalyst is too time-consuming to be practical 


36% 


45% 


In general, a lesson should fit on one page 


36% 


36% 


PLANalyst will be useful for everyday planning 


36% 


36% 


The program was cumbersome 


20% 


70% 


The type on the printouts was too small 


9% 


81% 


The printouts took up too much paper 


9% 


63% 



The results in the table above break down cleanly into three categories: at the top, desirable 
statements that most users agreed with; at the bottom, undesirable statements that users 
tended to disagree with, and the middle ground where targets of improvement can be 
identified. The statements shown in bold seem the most important to work on, and the next 
round of software refinement will focus on raising the ratings here. In practice, this means 
adding more categories for describing instructional events and adding rules to Anna's 
rulebase to smarten her advice. 
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One major step in this regard is the addition of a database of teaching strategies. Over 600 
ERIC citations have been identified as containing information on useful teaching techniques. 
In the final release version of PLANalyst, users will be able to access this database and print 
out strategies and citations appropriate to their present lesson. It is hoped that the database 
will be rich enough that even experienced teachers and trainers will be shown strategies that 
they would not have thought of, thus raising the perceived usefulness of Anna's advice and 
the global value of the tool. 
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Appendix A: Instructional Event Categories 

Prior Knowledge Activation 

Review Previous Learning 

Use Analogy 

Ask Review Questions 

Pre-Test 

Information Preview 

State Objectives Formally 
State Objectives Informally 
Show Final Goal 
Provide Overview 
Provide Background 
Teacher Pre-Questions 
Learner Pre-Questions 
Advance Organizer 

Motivation 

Novelty 

Arouse Uncertainty 
Evoke Mystery 
Cite Model 

Establish Future Relevance 
Establish Relevance to Past 
Raise Confidence ' 
Establish Reward 
Establish Credibility 
Establish Need 

Information Acquisition 

Give Definition 

Examples/Non-examples 

Definitions & Examples 

Procedure Description 

Procedure Demonstration 

Fact Presentation 

Problem Definition 

Discussion 

Socratic Dialog 

Analogies 

Anecdotes 

Inquiry Individual 

Inquiry - Group 

Peer Tutoring 

Teacher Mnemonic 

Learner Mnemonics 
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Practice & Feedback 
Individual Practice (Guided) 
Group Practice (Guided) 
Individual Practice (Independent) 
Group Practice (Independent) 
Group Feedback 
Peer Feedback 
Delayed Feedback 
Simulation 
Drill 

Brainstorming 
Role Play 
Q & A 
Debriefing 

Closure 

Reflection 
Student Self-Test 
Post-Test 
Teacher Summary 
Learner Summary 

Other 

Administration 

Assign Roles or Tasks 

Ice Breaker 

Social 

Break 

Rearrange Classroom 
Planning Time 
Session Evaluation 
Other (Instructional) 
Other (Non-Instructional) 



Appendix B: PLANalyst Evaluation Form 



Your Name (optional) 

Years of teaching experience ? Subject/Grade level? 

Thanks for giving PLANalyst a try. Your opinions about PLANalyst are very valuable to me 
as 1 refine its features in preparation for publication. Please give your thoughtful, frank and 
honest responses to the questions on this page. — Bernie Dodge 

For each statement, please circle the response that corresponds most closely with your 
opinion. 



SD = Strongly Disagree, D = Disagree, N = Neutral, A = Agree, SA = Strongly Agree 



PLANalyst will be useful for everyday lesson planning. 


SD 


D 


N 


A 


SA 


PLANalyst will be useful primarily for planning special or 
unusual lessons. 


SD 


D 


N 


A 


SA 


The Rearrange Events feature is useful 


SD 


D 


N 


A 


SA 


The Brainstorming feature is useful. 


SD 


D 


N 


A 


SA 


The program seemed to run slowly. 


SD 


D 


N 


A 


SA 


PLANalyst was easy to use. 


SD 


D 


N 


A 


SA 


The program was cumbersome. 


SD 


D 


N 


A 


SA 


It was difficult to find an appropriate category to describe 


SD 


D 


N 


A 


SA 


an 

event I wanted to include. 












The advice from Anna was useful and appropriate. 


SD 


D 


N 


A 


SA 


I'd use the "Lite" mode more often than the "Full" mode. 


SD 


D 


N 


A 


SA 


PLANalyst is too time-consuming to be practical. 


SD 


D 


N 


A 


SA 


There were enough different print out formats provided. 


SD 


D 


N 


A 


SA 


The printouts were legible and attractive. 


SD 


D 


N 


A 


SA 


The printouts took up too much paper. 


SD 


D 


N 


A 


SA 


In general, a lesson plan should fit on one page. 


SD 


D 


N 


A 


SA 


The type on the printouts was too small. 


SD 


D 


N 


A 


SA 


The time spent putting a lesson into PLANalyst form would 
be well spent. 


SD 


D 


N 


A 


SA 



Can you think of any categories of event types that should be added to the pop-up list? If so, 
please describe them. 



Can yo*' think of other features you'd like to see added? 



Did you experience any 'bugs' or flaky program behavior? If so, please describe. 



If you had to summarize your opinion of PLANalyst in just a few sentences, what would you 
say? 
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